enhance: Add compaction task slot usage logic #34581
Merged
+422
−32
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mergify
bot
added
dco-passed
|
@wayblink E2e jenkins job failed, comment |
|
/run-cpu-e2e |
wayblink
force-pushed
the
clustering-control
branch
2 times, most recently
from
287a20b to
7abe847
Compare
|
rerun ut |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #34581 +/- ##
========================================
Coverage 80.76% 80.77%
========================================
Files 1161 1161
Lines 141018 141148 +130
========================================
+ Hits 113898 114017 +119
- Misses 22755 22760 +5
- Partials 4365 4371 +6
|
wayblink
force-pushed
the
clustering-control
branch
from
7abe847 to
918d293
Compare
wayblink
changed the title
wayblink
force-pushed
the
clustering-control
branch
from
918d293 to
9b0e8db
Compare
wayblink
force-pushed
the
clustering-control
branch
2 times, most recently
from
3e0ff3c to
b4b9eaa
Compare
wayblink
force-pushed
the
clustering-control
branch
from
b4b9eaa to
afc4110
Compare
bigsheeper
reviewed
Jul 15, 2024
internal/datacoord/compaction.go
Outdated
| nodeID = id | ||
| maxSlots = slots | ||
| } | ||
| } | ||
| // update the input nodeSlots | ||
| nodeSlots[nodeID] = nodeSlots[nodeID] - acquireSlot |
There was a problem hiding this comment.
we should decrease the slots after nodeID meta saved
bigsheeper
reviewed
Jul 15, 2024
| @@ -558,7 +566,7 @@ func (t *clusteringCompactionTask) GetLabel() string { | |||
| } | |||
|
|
|||
| func (t *clusteringCompactionTask) NeedReAssignNodeID() bool { | |||
| return t.GetState() == datapb.CompactionTaskState_pipelining && t.GetNodeID() == 0 | |||
| return t.GetState() == datapb.CompactionTaskState_pipelining && (t.GetNodeID() == 0 || t.GetNodeID() == NullNodeID) | |||
There was a problem hiding this comment.
yes, I learn this from your pr
bigsheeper
reviewed
Jul 15, 2024
| dropped *typeutil.ConcurrentSet[string] // vchannel dropped | ||
| usingSlots *atomic.Int64 |
There was a problem hiding this comment.
Actually it is a double check. Executor can refuse new task even datacoord works unexpectedly.
bigsheeper
reviewed
Jul 15, 2024
| @@ -37,7 +39,7 @@ const ( | |||
|
|
|||
| type Executor interface { | |||
| Start(ctx context.Context) | |||
| Execute(task Compactor) | |||
| Execute(task Compactor) (bool, error) | |||
There was a problem hiding this comment.
yes, if slots are used up. return a DataNodeSlotExhausted error
wayblink
force-pushed
the
clustering-control
branch
4 times, most recently
from
3f61843 to
27d07dd
Compare
wayblink
force-pushed
the
clustering-control
branch
2 times, most recently
from
b7c26b4 to
28b8d7c
Compare
wayblink
changed the title
wayblink
force-pushed
the
clustering-control
branch
4 times, most recently
from
377995e to
a5a43c2
Compare
Signed-off-by: wayblink <[email protected]>
wayblink
force-pushed
the
clustering-control
branch
from
a5a43c2 to
eefcccb
Compare
sre-ci-robot
pushed a commit
that referenced
this pull request
Jul 18, 2024
issue: #34544 pr: #34581 --------- Signed-off-by: wayblink <[email protected]>
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: czs007, wayblink The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#34544